Apparatus and method for picking up article disposed in three-dimensional space using robot

ABSTRACT

An article pickup device configured so as to select a first and second three-dimensional points present in the vicinity of each other based on position information of the plurality of three-dimensional points acquired by a three-dimensional measurement instrument and image data acquired by a camera, acquire an image gradient information in a partial image region including points on an image corresponding to these three-dimensional points, judge whether the first and second three-dimensional points are present on the same article based on a position information of the three-dimensional points and the image gradient information, and add the first and second three-dimensional points to the same connected set when it is judged that the first and second three-dimensional points are present on the same article.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an article pickup apparatus and anarticle pickup method for recognizing a position and a posture of anarticle disposed in a three-dimensional space and for picking up therecognized article using a robot.

2. Description of the Related Art

As an apparatus of this type, conventionally, there is known anapparatus configured to recognize a position of an article by patternmatching with respect to a two-dimensional image obtained by imagingarticles disposed in a three-dimensional space using a camera or athree-dimensional point set obtained via measurement using athree-dimensional measurement instrument. Such an apparatus isdescribed, for example, in Japanese Laid-open Patent Publication No.2004-295223 (JP2004-295223A) and Japanese Laid-open Patent PublicationNo. 2011-179909 (JP2011-179909A). Further, there is also known anapparatus configured to extract an article in an image using image dataobtained by imaging articles disposed in a three-dimensional space usinga camera. An apparatus of this type is described, for example, inJapanese Laid-open Patent Publication No. 2010-039999 (JP2010-039999A).

In the apparatus described in JP2004-295223A, a two-dimensional modelpattern is prepared from a two-dimensional image obtained by previouslyimaging an article in a standard three-dimensional relative posture, andthen a plurality of transform two-dimensional model patterns areprepared by applying two-dimensional geometric transform to thetwo-dimensional model pattern to perform two-dimensional patternmatching with respect to a two-dimensional image of an article using theplurality of transform two-dimensional model patterns.

In the apparatus described in JP2011-179909A, while a three-dimensionalmodel pattern of an article is previously acquired from a CAD model orthe like, surfaces of articles in a three-dimensional space are measuredusing a three-dimensional measurement instrument and a three-dimensionalpoint set (a distance image) is acquired, and then the three-dimensionalpoint set is divided into partial regions surrounded by an edgeextracted from the three-dimensional point set. Then, initially, one ofthe partial regions is set as an article region, and both matchingprocessing of the three-dimensional model pattern for the article regionand update processing for adding another partial region to the articleregion are repeated to measure positions and postures of the articles.

In the apparatus described in JP2010-039999A, a color image and adistance image of a region including an article are acquired by animaging operation of a camera, and then the acquired images aredisplayed on a display part. When a part of the article in the displayedimage is taught by a user as a foreground region, a region other thanthe article is set as a background region based on color information ofa color image and distance information obtained from a distance image toextract the article in the image.

However, in the apparatuses described in JP2004-295223A andJP2011-179909A, it is necessary to previously prepare a two-dimensionalmodel pattern or a three-dimensional model pattern for each kind ofarticles, resulting in a need for time and effort. In particular, whenthere are a large number of different articles, it is necessary toprepare model patterns corresponding to the number of differentarticles, and therefore, much time and effort are necessary.

In the apparatus described in JP2010-039999A, in order to extract anarticle in an image, a teaching operation by a user is necessary,resulting in a need for time and effort. In particular, when there are alarge number of different articles, it is necessary to perform teachingoperations corresponding to the number of different articles, andtherefore, much time and effort are necessary.

SUMMARY OF THE INVENTION

An article pickup apparatus according to an aspect of the presentinvention includes: a robot including a hand capable of holding anarticle; a three-dimensional measurement instrument measuring surfacepositions of a plurality of articles disposed in a three-dimensionalspace to acquire position information of a plurality ofthree-dimensional points; a camera imaging an area including theplurality of articles to acquire image data; a connected set calculationpart determining a connected set made by connecting three-dimensionalpoints present in the vicinity of each other among the plurality ofthree-dimensional points, based on the position information of theplurality of three-dimensional points acquired by the three-dimensionalmeasurement instrument and the image data acquired by the camera; anarticle identification part identifying a position and posture of thearticle, based on the position information of the three-dimensionalpoints belonging to the connected set; a hand position posturecalculation part determining a hand position posture as a position and aposture of the hand capable of picking up the article identified by thearticle identification part; and a robot control part controlling therobot so as to move the hand to the hand position posture determined bythe hand position posture calculation part to pick up the article. Theconnected set calculation part includes: a three-dimensional pointselection part selecting a first three-dimensional point and a secondthree-dimensional point present in the vicinity of each other among theplurality of three-dimensional points acquired by the three-dimensionalmeasurement instrument; a gradient information acquisition partacquiring an image gradient information indicating a gradient state of asurface of the article in a partial image region including points on animage corresponding to respective the first three-dimensional point andthe second three-dimensional point, based on the image data acquired bythe camera; and a judgment part judging whether the firstthree-dimensional point and the second three-dimensional point arepresent on the same article, based on a position information of thefirst three-dimensional point, a position information of the secondthree-dimensional point, and the image gradient information acquired bythe gradient information acquisition part, and, when the judgment partjudges that the first three-dimensional point and the secondthree-dimensional point are present on the same article, the connectedset calculation part adds the first three-dimensional point and thesecond three-dimensional point to the same connected set.

Another aspect of the present invention is an article pickup method forpicking up an article disposed in a three-dimensional space using arobot including a hand capable of holding the article, the methodincluding: measuring, by a three-dimensional measurement instrument,surface positions of a plurality of articles disposed in thethree-dimensional space to acquire position information of a pluralityof three-dimensional points; imaging, by a camera, an area including theplurality of articles to acquire image data; determining a connected setmade by connecting the three-dimensional points present in the vicinityof each other among the plurality of three-dimensional points, based onthe position information of the plurality of three-dimensional pointsacquired by the three-dimensional measurement instrument and the imagedata acquired by the camera; identifying a position and posture of thearticle, based on the position information of the three-dimensionalpoints belonging to the connected set; determining a hand positionposture as a position and posture of the hand capable of picking up thearticle, the position and posture of which are identified; andcontrolling the robot so as to move the hand to the hand positionposture to pick up the article. The determining the connected setincludes: selecting a first three-dimensional point and a secondthree-dimensional point present in the vicinity of each other among theplurality of three-dimensional points acquired by the three-dimensionalmeasurement instrument; acquiring an image gradient informationindicating a gradient state of a surface of the article in a partialimage region including points on an image corresponding to respectivefirst three-dimensional point and second three-dimensional point, basedon the image data acquired by the camera; judging whether the firstthree-dimensional point and the second three-dimensional point arepresent on the same article, based on a position information of thefirst three-dimensional point, a position information of the secondthree-dimensional point, and the image gradient information; and addingthe first three-dimensional point and the second three-dimensional pointto the same connected set when it is judges that the firstthree-dimensional point and the second three-dimensional point arepresent on the same article.

BRIEF DESCRIPTION OF THE DRAWINGS

Objects, features, and advantages of the present invention will becomefurther apparent from the following description of an embodiment whentaken with the accompanying drawings in which:

FIG. 1 is a view illustrating a schematic configuration of an articlepickup apparatus according to one embodiment of the present invention;

FIG. 2 is a flowchart illustrating one example of processing executed ina robot control device of FIG. 1;

FIG. 3 is a view illustrating one example of a three-dimensional pointset acquired using a three-dimensional measurement instrument of FIG. 1;

FIG. 4 is a view illustrating one example of an image captured using acamera of FIG. 1;

FIG. 5 is a view illustrating one example of a connected set determinedfrom the three-dimensional point set of FIG. 3;

FIG. 6 is a conceptual view illustrating connected sets;

FIG. 7 is a view illustrating one example of a representative positionposture based on positions of three-dimensional points belonging to eachconnected set of FIG. 6;

FIG. 8 is a view illustrating one example of a hand position posturecorresponding to the representative position posture of FIG. 7;

FIG. 9 is a view illustrating one example of numbering for the handposition posture of FIG. 8;

FIG. 10 is a view illustrating one example of an operation of thearticle pickup apparatus according to the embodiment of the presentinvention;

FIG. 11 is a view illustrating one example of an operation following theoperation in FIG. 10;

FIG. 12 is a flowchart illustrating details of processing fordetermining the connected set of FIG. 2;

FIG. 13 is a flowchart illustrating details of vicinity judgmentprocessing of FIG. 12;

FIG. 14 is a view illustrating one example of three-dimensional pointsmeasured on a surface of an article;

FIG. 15 is a view illustrating one example of two-dimensional points onan image corresponding to the three-dimensional points of FIG. 14;

FIG. 16 is a view illustrating another example of two-dimensional pointson the image corresponding to the three-dimensional points of FIG. 14;and

FIG. 17 is a block diagram illustrating an internal configuration of therobot control device of FIG. 1.

DETAILED DESCRIPTION

Hereinafter, with reference to FIG. 1 to FIG. 17, an article pickupapparatus according to the embodiment of the present invention will bedescribed. FIG. 1 is a view illustrating a schematic configuration of anarticle pickup apparatus 10 according to one embodiment of the presentinvention. The article pickup apparatus 10 includes a three-dimensionalmeasurement instrument 11, a camera 12, a robot 13, and a robot controldevice 14 for controlling the robot 13 by connecting thethree-dimensional measurement instrument 11 and the robot 13. The robot13 includes a hand 15 mounted on a tip of an arm 13 a. A pallet 16 isdisposed sideward of the robot 13. Together therewith, FIG. 1illustrates an orthogonal three-axis coordinate system of X, Y, and Z.The Z-direction is a vertical direction, and the X-direction and theY-direction are horizontal directions.

On the pallet 16, a plurality of articles 20 are disposed. The articlepickup apparatus 10 of the present embodiment recognizes a position anda posture of an article 20 to be picked up while the plurality ofarticles 20 are disposed, picks up and holds the recognized article 20using the hand 15, and conveys the article 20 to a predeterminedposition by an operation of the robot 13. In FIG. 1, the plurality ofarticles 20 are illustrated so as to have the same shape as each other,but may be composed of a plurality of different articles having shapesdifferent from each other. In the following description, in some cases,the article 20 to be held by the hand 15 is expressed using a sign 21(refer to FIG. 11) for discrimination from another article on the pallet16.

The three-dimensional measurement instrument 11 is disposed above acenter portion of the pallet 16 and measures a surface of an exposedarticle 20 among the articles 20 disposed on the pallet 16 to acquireposition information (three-dimensional information) of a plurality ofthree-dimensional points. A measurement range of the three-dimensionalmeasurement instrument 11 needs to include the pallet 16 but anexcessively large measurement range decreases measurement resolution.Therefore, preferably, the measurement range is equivalent to anoccupied range of the pallet 16 and, for example, accords with theoccupied range of the pallet 16. In FIG. 1, the three-dimensionalmeasurement instrument 11 is fixed to a dedicated cradle 17 but may bemounted on a tip of the robot 13. The three-dimensional measurementinstrument 11 and the robot control device 14 are connected to eachother via a communication part such as a communication cable so as to becommunicable with each other.

As the three-dimensional measurement instrument 11, various non-contacttypes can be used. Such types include, for example, a stereotype usingtwo cameras, a scanning type using laser slit light, a scanning typeusing laser spot light, a type of projecting pattern light on an articleusing a device such as a projector, and a type of utilizing a flighttime from emission of light from a projector to incidence to a lightreceiver via reflection on an article surface.

The three-dimensional measurement instrument 11 expresses the acquiredthree-dimensional information as the format of a distance image or athree-dimensional map. The distance image is an image in whichthree-dimensional information is expressed as an image format, andexpresses a height of a position on an image or a distance from thethree-dimensional measurement instrument 11 using brightness or a colorof each pixel of the image. On the other hand, the three-dimensional mapis a map in which three-dimensional information is expressed as a set ofmeasured three-dimensional coordinate values (x, y, z). In the presentembodiment, each pixel in a distance image or a point havingthree-dimensional coordinate values in a three-dimensional map isreferred to as a three-dimensional point, and a set including aplurality of three-dimensional points is referred to as athree-dimensional point set. The three-dimensional point set is a set ofall the three-dimensional points measured using the three-dimensionalmeasurement instrument 11 and can be acquired using thethree-dimensional measurement instrument 11.

The camera 12 includes an imaging device such as a CCD, and is disposedabove a center portion of the pallet 16 to image articles 20 disposed onthe pallet 16. A photographing region of the camera 12 needs to includethe pallet 16 but an excessively large photographing region causes adecrease of imaging resolution. Therefore, preferably, a photographingregion is equivalent to an occupied range of the pallet 16 and, forexample, accords with the occupied range of the pallet 16. In FIG. 1,the camera 12 is fixed to the dedicated cradle 17 but may be mounted ona tip of the robot 13. The camera 12 and the robot control device 14 areconnected to each other via a communication part such as a communicationcable so as to be communicable with each other.

The camera 12 is previously calibrated, and use of calibration datamakes it possible to determine a correspondence relation between athree-dimensional point measured using the three-dimensional measurementinstrument 11 and a point (a two-dimensional point) on an image capturedusing the camera 12. In other words, it can be determined which point ina camera image a three-dimensional point corresponds to, whereby imagedata corresponding to a certain three-dimensional point becomesobtainable.

The hand 15 can pick up and also hold the article 20, and examples of aconfiguration of the hand capable of performing such operations include,for example, a suction nozzle, an attracting magnet, and a suction pador chuck. An operation of the robot 13 controls a position posture ofthe hand 15.

FIG. 2 is a flowchart illustrating one example of processing executed inthe robot control device 14 and specifically, one example of processingfor article pickup. An operation of the article pickup device 10 will bedescribed below with reference to the flowchart of FIG. 2 and drawingsassociated therewith.

Processing of FIG. 2 is started when, for example, a pickup startcommand of an article 20 is input by operating an operation switch notillustrated. Initially, surfaces of a plurality of articles 20 disposedin a three-dimensional space are measured using the three-dimensionalmeasurement instrument 11 and a three-dimensional point set 30 isacquired (step S1). FIG. 3 is a view illustrating one example of thethree-dimensional point set 30 acquired using the three-dimensionalmeasurement instrument 11 and three-dimensional points 31 configuringthe three-dimensional point set 30. In FIG. 3, the three-dimensionalpoints 31 are illustrated with black circles and the three-dimensionalpoint set 30 is illustrated as a region surrounded by a dotted lineincluding all the black circles.

Then, a region including a plurality of articles 20 is imaged using thecamera 12 and an image 40 is acquired (step S2). FIG. 4 is a schematicview illustrating one example of the image 40 captured using the camera12. FIG. 4 illustrates a pallet image 41 expressing the pallet 16 and anarticle image 42 expressing the article 20.

Then, at least one connected set 32 is determined from thethree-dimensional point set 30 (step S3). FIG. 5 is a view illustratingone example of the connected set 32 determined from thethree-dimensional point set 30. FIG. 5 illustrates the connected set 32as a region surrounded by a dotted line. In other words, FIG. 5illustrates two connected sets 32.

The connected set 32 referred to here is a partial set of thethree-dimensional point set 30, and when in the vicinity of an arbitrarythree-dimensional point (a first three-dimensional point) 31, anotherthree-dimensional point (a second three-dimensional point) 31 differentfrom the former three-dimensional point 31 is present, the connected set32 is a set where the first three-dimensional point 31 and the secondthree-dimensional point 31 are connected. FIG. 6 is a view illustratinga concept of the connected set 32. In FIG. 6, when a distance betweenthe first three-dimensional point 31 and the second three-dimensionalpoint 31 next to each other falls within a predetermined value, thefirst three-dimensional point 31 and the second three-dimensional point31 are connected with each other.

In other words, as illustrated in FIG. 6, when a plurality ofthree-dimensional points 31 (expressed by 311 to 317) are measured usingthe three-dimensional measurement instrument 11 and of these, 311 and312, 312 and 313, 313 and 314, and 315 and 316 are present within apredetermined distance, these points are connected with each other. Inthis case, 311 and 314 are also connected via 312 and 313 and therefore,311 to 314 configure the same connected set 321. On the other hand, 315and 316 are not connected to any of 311 to 314 and therefore, configureanother connected set 322. Since not being connected to anythree-dimensional point, 317 configures no connected set.

In the present embodiment, the connected set 32 is configured so that asingle connected set 32 corresponds to a single article 20, i.e., thearticle 20 and the connected set 32 correspond to each other on aone-to-one basis. Thereby, the article 20 is identified using theconnected set 32. In this case, using not only measurement data from thethree-dimensional measurement instrument 11 but also image data from thecamera 12, the connected set 32 is determined. Specific processing fordetermining the connected set 32 will be described later (FIG. 13).

Then, on the basis of positions of the three-dimensional points 31belonging to the same connected set 32, a representative positionposture 33 representing each connected set 32 is determined (step S4).The connected set 32 identifies a surface where the article 20 isexposed, and the representative position posture 33 refers to a positionand a posture representing the article 20. FIG. 7 is a view illustratingone example of the representative position posture 33 calculated basedon positions of the three-dimensional points 31 belonging to theconnected set 32. The representative position posture 33 is illustratedusing a pair of arrows 33 a and 33 b intersecting at a right angle,since the representative position posture 33 is expressed in anorthogonal coordinate system. In FIG. 7, the representative positionposture 33 is illustrated using the two arrows 33 a and 33 b but therepresentative position posture 33 is not present in a two-dimensionalspace but in a three-dimensional space.

There are several methods for determining the representative positionposture 33. Initially, as a first example, there is available a methodin which a gravity center position of three-dimensional points 31belonging to the connected set 32 and a predetermined posture (forexample, a posture where the arrow 33 a is directed upward in a verticaldirection) are combined to obtain the representative position posture33. To calculate the gravity center position, all the three-dimensionalpoints 31 belonging to the connected set 32 are employable, orthree-dimensional points 31 selected by separately introducingprocessing such as outlier countermeasures are also employable. For theoutlier countermeasures, for example, initially, a gravity centerposition is determined using all the three-dimensional points 31belonging to the connected set 32 for gravity center calculation, andwhen three-dimensional points 31 having at least a predetermined valueof a distance from the gravity center position exist among thethree-dimensional points 31 used for gravity center calculation,three-dimensional points 31 of a predetermined ratio are eliminated fromthe three-dimensional points 31 used for gravity center calculation indescending order of the distance from the gravity center position. Then,the remaining three-dimensional points 31 are used for gravity centercalculation to recalculate a gravity center position. This processingmay be repeated until all the three-dimensional points 31 used forgravity center calculation fall within the predetermined distance fromthe gravity center position.

As a second example, there is available a method in which a rectangle (acircumscribed rectangle) circumscribed to three-dimensional points 31belonging to the connected set 32 is determined to allow a positionposture of the center of the circumscribed rectangle to be therepresentative position posture 33. To determine the circumscribedrectangle, initially, using all the three-dimensional points 31 includedin the connected set 32, a plane is determined in a three-dimensionalspace and then all the three-dimensional points 31 included in theconnected set 32 are projected onto this plane. Then, a convex polygoninternally including all the projected three-dimensional points 31 iscalculated and a rectangle circumscribed to the calculated convexpolygon is determined. The plane may be determined by a least-squaresmethod using all the three-dimensional points 31 belonging to theconnected set 32 or by separately introducing any processing for outliercountermeasures. As a method for outlier countermeasures, severalmethods such as an M-estimation method, RANSAC, LMedS, and Houghtransform are available. As a method for calculating a convex polygon,Andrew's Algorithm method and the like can be employed. As a method forcalculating a rectangle circumscribed to a convex polygon, a rotatingcalipers method and the like can be employed. FIG. 7 illustrates a casewhere the representative position posture 33 is determined based on thefirst example.

Then, a hand position posture 34 corresponding to each representativeposition posture 33 is determined (step S5). FIG. 8 is a viewillustrating one example of the hand position posture 34 correspondingto the representative position posture 33. The hand position posture 34is illustrated using a pair of arrows 34 a and 34 b intersecting atright angles in the same manner as for the representative positionposture 33.

As a method for determining a position (an intersection of the arrows 34a and 34 b) and a posture (directions of the arrows 34 a and 34 b) ofthe hand position posture 34, several methods are available for each.Regarding the position, a method in which, for example, a position ofthe representative position posture 33 is directly assigned as aposition of the hand position posture 34 can be employed. As anotherexample, there is also a method in which a position moved by apredetermined length in a direction of a predetermined coordinate axis35 (e.g., Z-axis) from a position of the representative position posture33 is specified as a position of the hand position posture 34. FIG. 8illustrates a position based on the latter example. Regarding theposture, a method in which, for example, a posture of the representativeposition posture 33 is directly assigned as a posture of the handposition posture 34 is available. As another example, there is also amethod in which a posture rotated at a predetermined angle around thepredetermined coordinate axis 35 of the representative position posture33 is specified as a posture of the hand position posture 34. FIG. 8illustrates a posture based on the former example.

Then, the respective hand position postures 34 are numbered as P1, P2, .. . , Pn (step S6). Here, n represents the number of hand positionpostures 34. FIG. 9 is a view illustrating the numbered hand positionpostures 34 and numbering is performed in descending order of acoordinate value with respect to the predetermined coordinate axis 35,i.e., in order from a value at a higher position. As illustrated in FIG.9, when coordinate values with respect to the coordinate axis 35 are thesame, numbering is performed in descending order of a coordinate valuewith respect to a predetermined coordinate axis 36 (for example, X-axis)at right angles to the coordinate axis 35. In FIG. 9, n=2.

Then, an initial value is provided for a variable k having a naturalnumber. In other words, processing for k←1 is executed (step S7). Thevariable k is used for specifying a number for the hand position posture34.

Then, as illustrated in FIG. 10, a control signal is output to a robotdriving actuator (an electric motor) and the hand 15 is moved to a handposition posture Pk by an operation of the robot 13 (step S8). Regardingan initial value of the variable: k=1, Pk=P1.

Then, a control signal for holding the article 20 is output to a handdriving actuator and as illustrated in FIG. 11, an article 21 is held bya lower end surface of the hand 15 (step S9). When, for example, thehand 15 includes a suction nozzle, a vacuum pump is operated to suck andhold the article 21 by a suction force. When the hand 15 includes anattracting magnet, current is allowed to flow in an electromagnetic coiland the magnet is operated to attract and hold the article 21 by amagnetic force. When the hand 15 includes a chuck, the article 21 isheld by opening or closing the chuck.

Then, it is judged whether the hand 15 has succeeded in holding thearticle 21 (step S10). Regarding this judgment, when the hand 15includes a suction nozzle, it may be judged whether the hold has beensuccessfully performed, according to a change of a flow rate or pressureof air during suction. When the hand 15 includes an attracting magnet,it is possible to judge whether the article 21 is present using aproximity sensor and then to judge whether a hold has been successfullyperformed, according to the presence or absence thereof. When the hand15 includes a chuck, it is possible to judge whether a hold has beensuccessfully performed by confirming an opening or closing state of thechuck using an opening/closing confirmation sensor. When it is judgedthat the hold has been successfully performed, the processing moves tostep S11. When it is judged that the hold has not been successfullyperformed, the processing moves to step S12.

In step S12, it is judged whether the variable k is smaller than n. Thisjudgment is a judgment whether any hand position posture which the hand15 has not reached yet exists among n (2 in FIG. 11) hand positionpostures 34. When k<n is judged, the hand 15 has not reached yet a handposition posture Pk+1 and therefore, processing for k←k+1 is executed(step S13) and then the processing returns to step S8. When it is judgedthat k<n is not satisfied in step S12, the hand 15 has reached all nhand position postures 34 and therefore, the processing returns to stepS1.

In step S11, a control signal is output to the robot driving actuator tomove the hand 15 to a predetermined position. Thereby, the article 21 isconveyed to the predetermined position by an operation of the robot 13.Thereafter, a control signal is output to the hand driving actuator toremove the article 21 from the hand 15. This indicates the end of onecycle of the processing.

In the above description, processing in the robot control device 14acquires a three-dimensional point set 30 including a plurality ofthree-dimensional points 31 by measuring surface positions of aplurality of articles 20 using the three-dimensional measurementinstrument 11 (step S1); determines a connected set 32 made byconnecting three-dimensional points 31 present in the vicinity of eachother from the three-dimensional point set 30 (step S3); determines aposition posture (a hand position posture 34) of the hand 15 capable ofpicking up the article 20 based on position information of thethree-dimensional points 31 belonging to the connected set 32 (step S4and step S5); and further controls the robot 12 so as to pick up thearticle 20 on the pallet 16 by moving the hand 15 to the determined handposition posture 34 (step S8 to step S11).

The connected set 32 reflects a position and a posture (inclination) ofan article surface and therefore, use of the connected set 32 makes itpossible to identify a position and a posture of the article 20 withoutpattern matching, a teaching operation by a user, and the like.Therefore, a model pattern of the article 20 does not need to beprepared, and regarding even a large number of different types ofarticles 20 differing in shape, it is possible to easily recognizepositions and postures thereof to hold the articles 20. Further, alsoregarding a new kind of article 20 added, it is possible to easilyrecognize a position and a posture thereof without addition of a modelpattern, a teaching operation by a user, and the like.

When a plurality of articles 20 having the same shape are disposed nextto each other, there is a possibility that the first three-dimensionalpoint 31 and the second three-dimensional point 31 are measured onarticles 20 different from each other. At that time, upon calculating adistance between the first three-dimensional point 31 and the secondthree-dimensional point 31, the distance falling within a predeterminedvalue, when the first three-dimensional point 31 and the secondthree-dimensional point 31 are configured so as to belong to the sameconnected set 32 assuming that the first three-dimensional point 31 andthe second three-dimensional point 31 are present in the vicinity ofeach other, the connected set 32 is configured across a plurality ofarticles 20. In order to avoid this problem, in the present embodiment,the connected set 32 is calculated as described below.

FIG. 12 and FIG. 13 each are a flowchart illustrating one example ofprocessing (connected set calculation processing) for determining theconnected set 32 that is a characteristic feature of the presentembodiment, i.e., a flowchart specifically illustrating processing instep S3 of FIG. 2.

Initially, step S21 of FIG. 12 allocates all the three-dimensionalpoints 31 belonging to the three-dimensional point set 30 with a labelnumber 0 indicating no belongingness to any connected set 32 as aninitial label number. In the following description, a three-dimensionalpoint 31 allocated with a label number j which is a natural number isexpressed using 31(j). The label number j is a number allocatedcorresponding to the connected set 32, and when the same label number jwhich is not 0 is allocated, belonging to the same connected set 32 ismeant. Then, in step S22, in order to determine a first connected set32, the label number j is set to be 1 (j←1).

Then, step S23 selects an arbitrary three-dimensional point 31(0) havinga label number of 0 which is a three-dimensional point 31 belonging tothe three-dimensional point set 30. Step S24 judges whether thethree-dimensional point 31(0) having a label number of 0 has beenselected, and when a judgment of Yes is made, the processing moves tostep S25. When the three-dimensional point 31(0) has not been selected,all the three-dimensional points 31 belonging to the three-dimensionalpoint set 30 belong to any one of the connected sets 32. In this case, ajudgment of No is made in step S24 to end the connected set calculationprocessing and then the processing moves to step S4 of FIG. 2.

Step S25 prepares a list Lj for storing a three-dimensional point 31(j)having a label number of j. Step S26 allocates the three-dimensionalpoint 31(0) selected in step S24 with a label number j and then adds thethree-dimensional point 31(j) to the list Lj. In step S27, an initialvalue 1 is provided for a variable m having a natural number value(m←1). The variable m refers to a number specifying thethree-dimensional point 31(j) included in the list Lj. It is assumedthat in the list Lj, added three-dimensional points 31(j) are lined upin order of addition.

In step S30, the following processing (vicinity judgment processing) isexecuted: i.e., it is judged whether in the vicinity of an mththree-dimensional point 31(j) of the list Lj, a three-dimensional point31(0) having a label number of 0 present on the same article 20 exists,and the three-dimensional point 31 (0) judged to exist is added to thelist Lj.

FIG. 13 is a flowchart illustrating details of the vicinity judgmentprocessing of step S30. Step S30A selects the mth three-dimensionalpoint 31(j) of the list Lj. Using the image 40 acquired in step S2, stepS30B calculates a two-dimensional point 51(j) on the image 40corresponding to the three-dimensional point 31(j) selected in stepS30A. In this case, use of calibration data previously set for thecamera 12 makes it possible to calculate the two-dimensional point 51(j)corresponding to the three-dimensional point 31(j).

Step S30C calculates a vicinity three-dimensional point set which is aset determined by collecting all the three-dimensional points (vicinitythree-dimensional points 310(0)) which are three-dimensional points31(0) having a label number of 0 and are present in the vicinity of thethree-dimensional point 31(j) selected in step S30A. The vicinitythree-dimensional point 310(0) is, for example, a point having anx-coordinate and a y-coordinate having a difference by at most apredetermined value from an x-coordinate and a y-coordinate of theselected three-dimensional point 31(j), respectively. It is possiblethat a point having an x-coordinate, a y-coordinate, and a z-coordinatehaving a difference by at most a predetermined value from anx-coordinate, a y-coordinate, and a z-coordinate of the selectedthree-dimensional point 31(j), respectively, is designated as thevicinity three-dimensional point 310(0) or a point having a Euclideandistance falling within a predetermined value is designated as thevicinity three-dimensional point 310(0). The vicinity three-dimensionalpoint 310(0) is a three-dimensional point 31(j) predicted to be presenton the same article and becomes a candidate of the three-dimensionalpoint 31(j) having a label number of j.

Step S30D numbers all the vicinity three-dimensional points 310(0)belonging to a vicinity three-dimensional point set as 310(1), 310(2), .. . , 310(n). In step 30E, an initial value of 1 is provided for avariable i having a natural number value. In other words, processing fori←1 is executed. Step S30F selects a numbered vicinity three-dimensionalpoint 310(i). Using calibration data of the camera 12, step S30Gcalculates a vicinity two-dimensional point 510(i) which is a point onthe image 40 corresponding to the vicinity three-dimensional point 310(i).

Step S30H acquires image gradient information of a region (a partialimage region) including the two-dimensional point 51(j) on the image 40calculated in step S30B and the two-dimensional point 510(i) on theimage 40 calculated in step S30G. The partial image region refers to aregion present within a predetermined distance from a line segmentconnecting the two-dimensional point 51(j) and the two-dimensional point510(i), for example. The image gradient information refers toinformation of a pixel value having at least a certain contrast(brightness) among filtered images determined by filtering a partialimage region using a Sobel filter or the like, for example. In otherwords, a physical amount indicating a sudden change in brightnessbetween pixels next to each other is acquired as image gradientinformation. It is also possible to use, as image gradient information,a difference between a pixel value having a minimum brightness and apixel value having a maximum brightness in the partial image region.Such a change amount in image contrast makes it possible to identify agradient state of a surface of the article 20.

Step S30I judges whether both the three-dimensional point 31(j) selectedin step S30A and the vicinity three-dimensional point 310(i) selected instep S30F are present on the same article 20, based on the imagegradient information acquired in step S30H. This judgment is made bydetermining whether, for example, a distance between thethree-dimensional point 31(j) and the vicinity three-dimensional point310(i) is equal to or less than a predetermined value Δd and also achange amount in image contrast in the partial image region is equal toor less than a predetermined value. Regarding the distance between thethree-dimensional point 31(j) and the vicinity three-dimensional point310(i), it is possible, for example, to previously determine apredetermined distance Δd for each of predetermined coordinate systemsfrom one-dimension to three-dimension and then to judge whether adistance between the three-dimensional point 31(j) and the vicinitythree-dimensional point 310(i) in every predetermined coordinate systemis equal to or less than the predetermined value Δd.

When a judgment of Yes is made in step S30I, the processing moves tostep S30J, and when a judgment of No is made, the processing passes stepS30J and then moves to step 30K. Step S30J allocates a label number j tothe vicinity three-dimensional point 310(i) selected in step S30F andadds the vicinity three-dimensional point 310(i) to the bottom of thelist Lj as the three-dimensional point 31(j). Step S30K adds 1 to thevariable i (i←i+1). Step S30L judges whether the judgment of step S30Ihas been made for all the vicinity three-dimensional points 310(1) to310(n), i.e., whether the variable i is larger than n. When a judgmentof No is made in step S30L, the processing returns to step S30F and thesame processing as described above is repeated. When a judgment of Yesis made in step S30L, the vicinity judgment processing (step S30) isended and the processing moves to step S31 of FIG. 12.

Step S31 adds 1 to the variable m (m←m+1). Step S32 judges whether avalue of m is larger than the number (element number N) ofthree-dimensional points 31(j) stored in the list Lj. The case where mis larger than the element number N indicates that the vicinity judgmentprocessing for all N three-dimensional points 31(j) stored in the listLj has been ended and that three-dimensional points present in thevicinity of the three-dimensional points 31(j) in the list Lj have beenalready stored in the same list Lj. Therefore, processing for adding athree-dimensional point 31(j) to the list Lj is ended and then theprocessing moves to step S33. In cases other than the above case, thevicinity judgment processing for all the three-dimensional points 31(j)in the list Lj has not been ended and therefore, the processing moves tostep S30 to repeat processing for adding a three-dimensional point 31(j)to the list Lj.

Step S33 adds 1 to the label number j (j←j+1) and the processing returnsto step S23. Thereafter, the same processing as step S23 to step S32 isrepeated to determine a connected set 32 corresponding to the next labelnumber j.

The connected set calculation processing described above will bespecifically described with reference to FIG. 6 and FIG. 14 to FIG. 16.At the start of connected set calculation processing, all thethree-dimensional points 311 to 317 illustrated in FIG. 6 do not belongto the connected set 32 and then a label number for thethree-dimensional points 311 to 317 is 0 (step S21). In order to preparea connected set 32 having a label number of 1 from this state, forexample, the three-dimensional point 314 is selected (step S23) and thethree-dimensional point 314 is allocated with a label number 1 (314(1)),and thereafter, the three-dimensional point 314 is stored as a first ofa list L1 having a label number of 1 (step S26).

Then, it is judged whether the three-dimensional point 31(0) having alabel number of 0 is present in the vicinity of the firstthree-dimensional point 314 of the list L1 and on the same article asthe article 20 where the three-dimensional point 314 has been measured(step S30). It is assumed that, for example, as illustrated in FIG. 14,the three-dimensional points 313 to 315 are measured on a surface of thearticle 20; and 313 and 314, and 314 and 315 are separated from eachother by the same distance which is at most the predetermined distanceΔd, respectively. At that time, as a two-dimensional point 51 on theimage 40, two-dimensional points 513 to 515 corresponding to therespective three-dimensional points 313 to 315 as illustrated in FIG. 15are calculated (step S30G). The two-dimensional points 513 and 514correspond to the three-dimensional points 313 and 314 on the samearticle, respectively, and therefore, a change amount in image contrastin a partial image region 521 including the two-dimensional points 513and 514 is equal to or less than a predetermined value. Therefore, it isjudged that the three-dimensional points 313 and 314 are present on thesame article (step S30I) and the three-dimensional point 313 isallocated with a label number 1 (313(1)) to be added as a second of thelist L1 (step S30J). Thereby, the three-dimensional point 314 and thethree-dimensional point 315 are connected, and the element number N ofthe list L1 becomes 2.

On the other hand, the two-dimensional points 514 and 515 correspond tothe three-dimensional points 314 and 315 on articles different from eachother, respectively, and therefore, as illustrated in FIG. 16, a changeamount in image contrast in a partial image region 522 including thetwo-dimensional points 514 and 515 exceeds the predetermined value.Therefore, it is judged that the three-dimensional points 314 and 315are not present on the same article, and then the three-dimensionalpoint 315 is not added to the same list L1 as for the three-dimensionalpoint 314. When vicinity judgment processing for all the vicinitythree-dimensional points 310(0) present in the vicinity of thethree-dimensional point 314 is ended, m becomes 2 (<N) (step S31), andin the same manner as described above, it is judged whether thethree-dimensional point 31(0) having a label number of 0 is present inthe vicinity of the second three-dimensional point 313 of the list L1 onthe same article as the article 20 where the three-dimensional point 313is measured (step S30).

When m is larger than the element number N, preparation of the list L1having a label number of 1 is ended and the label number is set as 2(step S33) to repeat the same processing. In repetition processing, forexample, a three-dimensional point 315 having a label number of 0 isallocated with a label number 2, and three-dimensional points 315(2) and316(2) are added to a list L2; and a three-dimensional point 317 havinga label number of 0 is allocated with a label number 3 and athree-dimensional point 317(3) is added to a list L3. Thereby, thethree-dimensional point 31 having a label number of 0 becomes absent andtherefore, a judgment of No is made in step S24 and then the connectedset calculation processing is ended.

The present embodiment makes it possible to achieve the followingoperations and effects.

(1) Processing in the robot control device 14 selects a firstthree-dimensional point 31 and a second three-dimensional point 31present in the vicinity of each other from a plurality ofthree-dimensional points 31 acquired using the three-dimensionalmeasurement instrument 11, (step S30A and step S30F); acquires a imagegradient information indicating a gradient state of a surface of thearticle 20 in a partial image region including points (two-dimensionalpoints 51) on an image corresponding to the respective firstthree-dimensional point 31 and second three-dimensional point 31, basedon image data acquired using the camera 12 (step S30H); judges whetherthe first three-dimensional point 31 and the second three-dimensionalpoint 31 are present on the same article, based on three-dimensionalposition information of the first three-dimensional point 31 and thesecond three-dimensional point 31 and the acquired image gradientinformation (step S30I); and further allocates the same label number jto these three-dimensional points 31 when it is judged that the firstthree-dimensional point 31 and the second three-dimensional point 31 arepresent on the same article to include the three-dimensional points 31allocated with the number in the same connected set 32.

Therefore, even when, for example, a plurality of articles 20 having thesame shape are disposed next to each other, the connected set 32 is notcalculated across the plurality of articles 20 and therefore, a singleconnected set 32 can be allowed to correspond to a single article 20.Accordingly, the connected set 32 makes it possible to accuratelyrecognize the article 20 and then correctly perform an article pickupoperation.

(2) Processing in the robot control device 14 calculates two-dimensionalpoints 51 on an image corresponding to the respective firstthree-dimensional point 31 and second three-dimensional point 31 (stepS30B and step 30G); and identifies a partial image region by thesetwo-dimensional points 51 and judges that the first three-dimensionalpoint 31 and the second three-dimensional point 31 are present on thesame article when a gradient state in the partial image region does notchange (step S30I). Therefore, it can be completely determined whetherthe first three-dimensional point 31 and the second three-dimensionalpoint 31 are present on the same article.

(3) In this case, the respective two-dimensional points 51 are acquiredbased on calibration data of the camera 12 and therefore, atwo-dimensional point 51 on an image corresponding to thethree-dimensional point 31 can be easily determined.

(4) Information of a change amount in image contrast in the partialimage region is used as image gradient information and therefore, agradient state of a surface of the article 20 can be accuratelydetermined.

(5) When a distance between the first three-dimensional point 31 and thesecond three-dimensional point 31 measured using the three-dimensionalmeasurement instrument 11 is equal to or less than a predetermined valueΔd and also a change amount in image contrast in the partial imageregion is equal to or less than a predetermined value, it is judged thatthe first three-dimensional point 31 and the second three-dimensionalpoint 31 are present on the same article. Thereby, it can be accuratelyjudged whether a plurality of three-dimensional points 31 are present onthe same article.

(6) Processing in the robot control device 14 determines arepresentative position posture 33 which is a position and a posturerepresenting a connected set 32 based on positions of three-dimensionalpoints 31 belonging to the connected set 32 (step S4); and determines ahand position posture 34 corresponding to this representative positionposture 33 (step S5). Thereby, a position relation between the article20 and the hand 15 can be appropriately set according to a type of thehand 15 and others.

(7) Processing in the robot control device 14 makes it possible tostably hold the article 20 using the hand 15 when determining therepresentative position posture 33 by combining a gravity centerposition of all the three-dimensional points 31 belonging to theconnected set 32 and a predetermined posture in this gravity centerposition.

(8) Processing in the robot control device 14 makes it possible tostably calculate the center of the article 20 as the representativeposition posture 33 and then stably hold the article 20 using the hand15 even upon existence of a deficit in a three-dimensional point 31 onan article measured using the three-dimensional measurement instrument11, when determining the representative position posture 33 by combininga center position of a circumscribed rectangle including all thethree-dimensional points 31 belonging to the connected set 32 and apredetermined posture.

The article pickup method may be configured in any manner when thefollowing is satisfied: when the connected set 32 is determined, a firstthree-dimensional point 31 and a second three-dimensional point 31present in the vicinity of each other are selected from a plurality ofthree-dimensional points 31 acquired using the three-dimensionalmeasurement instrument 11; an image gradient information indicating agradient state of a surface of the article 20 is acquired in a partialimage region including points 51 on an image corresponding to therespective first three-dimensional point 31 and second three-dimensionalpoint 31, based on image data acquired using the camera 12; it is judgedwhether the first three-dimensional point 31 and the secondthree-dimensional point 31 are present on the same article, based onthree-dimensional position information of the first three-dimensionalpoint 31 and the second three-dimensional point 31 and the acquiredimage gradient information; and the first three-dimensional point 31 andthe second three-dimensional point 31 are included in the same connectedset when it is judged that the first three-dimensional point 31 and thesecond three-dimensional point 31 are present on the same article.

FIG. 17 is a block diagram illustrating an internal configuration of therobot control device 14 of FIG. 1. The robot control device 14 includesa connected set calculation part 141, an article identification part142, a hand position posture calculation part 143, and a robot controlpart 144. The connected set calculation part 141 includes athree-dimensional point selection part 141A, a gradient informationacquisition part 141B, and a judgment part 141C.

In the present embodiment, a vicinity three-dimensional point 310present within a predetermined distance from the first three-dimensionalpoint 31 is calculated as the second three-dimensional point 31 (stepS30C), but the three-dimensional point selection part 141A may beconfigured in any manner, when the first three-dimensional point 31 andthe second three-dimensional point 31 present in the vicinity of eachother are selected. In the present embodiment, information of a changeamount in image contrast in a partial image region is acquired as animage gradient information (step S30H), but when a gradient state of asurface of the article 20 is indicated, any the image gradientinformation is employable and a configuration of the gradientinformation acquisition part 141B is not limited to the configurationdescribed above. In the present embodiment, by judging whether a changeamount in image contrast in a partial image region is equal to or lessthan a predetermined value (step S30I), it is judged that the firstthree-dimensional point 31 and the second three-dimensional point 31 arepresent on the same article, but a configuration of the judgment part141C is not limited to the configuration described above, when it isjudged whether the first three-dimensional point 31 and the secondthree-dimensional point 31 are present on the same article based onthree-dimensional position information of the first three-dimensionalpoint 31 and the second three-dimensional point 31 and image gradientinformation acquired using the gradient information acquisition part141B. In other words, the connected set calculation part 141 may beconfigured in any manner, when a connected set 32 made by connectingthree-dimensional points 31 present in the vicinity of each other isdetermined from a plurality of three-dimensional points 31 acquiredusing the three-dimensional measurement instrument 11 based onthree-dimensional position information of three-dimensional pointsacquired using the three-dimensional measurement instrument 11 and imagedata acquired using the camera 12.

In the present embodiment, the representative position posture 33representing a connected set 32 is calculated based on positioninformation of three-dimensional points 31 belonging to the connectedset 32 (step S3), but when a position and a posture of the article 20expressed by the connected set 32 are identified, a configuration of thearticle identification part 142 is not limited to the configurationdescribed above. A position and a posture representing the connected set32 also correspond to a position and a posture representing the article20, and identification of a position and a posture of the article 20refers to identification of a placement of the article 20 by determininga position and a posture representing articles. In the presentembodiment, the hand position posture 34 is calculated from therepresentative position posture 33 (step S4), but when a hand positionposture 34 capable of picking up an article 20 identified as therepresentative position posture 33 is determined, a configuration of thehand position posture calculation part 143 is not limited to theconfiguration described above. When the robot 13 is controlled so as topick up the article 20 by moving the hand 14 to the hand positionposture 34, the robot control part 144 may be configured in any manner.

It is possible to optionally combine the present embodiment with onemodified example or a plurality of modified examples.

According to the present invention, when a position and a posture of anarticle disposed in a three-dimensional space are recognized, an imagegradient information is acquired based on image data acquired using acamera; a connected set of a plurality of three-dimensional points isdetermined using three-dimensional position information ofthree-dimensional points measured using a three-dimensional measurementinstrument and image gradient information; and a position and a postureof an article is identified using the connected set. Therefore, aposition and a posture of an article can be easily recognized withoutpreparing a model pattern for the article and performing a teachingoperation for a user.

The present invention has been described in association with thepreferred embodiment, but it should be understood by those skilled inthe art that various corrections and modifications may be made withoutdeparting from the disclosed scope of the claims to be described later.

1. An article pickup device, comprising: a robot including a handcapable of holding an article; a three-dimensional measurementinstrument measuring surface positions of a plurality of articlesdisposed in a three-dimensional space to acquire a position informationof a plurality of three-dimensional points; a camera imaging an areaincluding the plurality of articles to acquire image data; a connectedset calculation part determining a connected set made by connectingthree-dimensional points present in the vicinity of each other among theplurality of three-dimensional points, based on the position informationof the plurality of three-dimensional points acquired by thethree-dimensional measurement instrument and the image data acquired bythe camera; an article identification part identifying a position andposture of the article, based on the position information of thethree-dimensional points belonging to the connected set; a hand positionposture calculation part determining a hand position posture as aposition and posture of the hand capable of picking up the articleidentified by the article identification part; and a robot control partcontrolling the robot so as to move the hand to the hand positionposture determined by the hand position posture calculation part to pickup the article, wherein the connected set calculation part comprises: athree-dimensional point selection part selecting a firstthree-dimensional point and a second three-dimensional point present inthe vicinity of each other among the plurality of three-dimensionalpoints acquired by the three-dimensional measurement instrument; agradient information acquisition part acquiring an image gradientinformation indicating a gradient state of a surface of the article in apartial image region including points on an image corresponding torespective the first three-dimensional point and the secondthree-dimensional point, based on the image data acquired by the camera;and a judgment part judging whether the first three-dimensional pointand the second three-dimensional point are present on the same article,based on a position information of the first three-dimensional point, aposition information of the second three-dimensional point, and theimage gradient information acquired by the gradient informationacquisition part, and wherein, when the judgment part judges that thefirst three-dimensional point and the second three-dimensional point arepresent on the same article, the connected set calculation part adds thefirst three-dimensional point and the second three-dimensional point tothe same connected set.
 2. The article pickup device according to claim1, wherein the gradient information acquisition part acquires points inthe image corresponding to the first three-dimensional point and thesecond three-dimensional point respectively, as a first two-dimensionalpoint and a second two-dimensional point specifying the partial imageregion, and the judgment part judges that the first three-dimensionalpoint and the second three-dimensional point are present on the samearticle, when a distance between the first three-dimensional point andthe second three-dimensional point is equal to or less than apredetermined value and a change amount of contrast of the image in thepartial image region is equal to or less than a predetermined value. 3.The article pickup device according to claim 2, wherein the gradientinformation acquisition part acquires the first two-dimensional pointand the second two-dimensional point based on calibration data of thecamera.
 4. The article pickup device according to claim 1, wherein thearticle identification part determines a representative position postureas a position and posture representing the connected set, based on theposition information of the three-dimensional points belonging to theconnected set, and the hand position posture calculation part determinesthe hand position posture corresponding to the representative positionposture.
 5. The article pickup device according to claim 4, wherein thearticle identification part determines the representative positionposture by combining a gravity center position of all thethree-dimensional points belonging to the connected set and apredetermined posture in the gravity center position.
 6. The articlepickup device according to claim 4, wherein the article identificationpart obtains the representative position posture by combining a centerposition of a circumscribed rectangle including all thethree-dimensional points belonging to the connected set and apredetermined posture.
 7. The article pickup device according to claim1, wherein the three-dimensional measurement instrument includes thecamera.
 8. An article pickup method for picking up an article disposedin a three-dimensional space, using a robot including a hand capable ofholding the article, the method comprising: measuring, by athree-dimensional measurement instrument, surface positions of aplurality of articles disposed in the three-dimensional space to acquireposition information of a plurality of three-dimensional points;imaging, by a camera, an area including the plurality of articles toacquire image data; determining a connected set made by connecting thethree-dimensional points present in the vicinity of each other among theplurality of three-dimensional points, based on the position informationof the plurality of three-dimensional points acquired by thethree-dimensional measurement instrument and the image data acquired bythe camera; identifying a position and posture of the article, based onthe position information of the three-dimensional points belonging tothe connected set; determining a hand position posture as a position andposture of the hand capable of picking up the article, the position andposture of which are identified; and controlling the robot so as to movethe hand to the hand position posture to pick up the article, whereinthe determining the connected set includes: selecting a firstthree-dimensional point and a second three-dimensional point present inthe vicinity of each other among the plurality of three-dimensionalpoints acquired by the three-dimensional measurement instrument;acquiring image gradient information indicating a gradient state of asurface of the article in a partial image region including points on animage corresponding to respective first three-dimensional point andsecond three-dimensional point, based on the image data acquired by thecamera; judging whether the first three-dimensional point and the secondthree-dimensional point are present on the same article, based on aposition information of the first three-dimensional point, a positioninformation of the second three-dimensional point, and the imagegradient information; and adding the first three-dimensional point andthe second three-dimensional point to the same connected set when it isjudged that the first three-dimensional point and the secondthree-dimensional point are present on the same article.